Transmitting method and receiving method

ABSTRACT

A transmitting method for transmitting encoded data obtained by hierarchically encoding a video image includes: generating an encoded stream that includes time information and the encoded data, the time information indicating a time at which decoding or displaying processing of the encoded data is performed, and transmitting the generated encoded stream, wherein the encoded data includes a plurality of sets each including a plurality of access units and the time information includes first time information which indicates a time at which the processing performed on a first access unit of the first set is performed and which is based on a reference clock, and second time information used to specify a time at which the processing performed on a second access unit of the second set is performed and which is based on the reference clock.

BACKGROUND

1. Technical Field

The present disclosure relates to a data transmitting method and a datareceiving method.

2. Description of the Related Art

Conventionally, a technique of transmitting encoded data inpredetermined transmission formats is known. The encoded data isgenerated by encoding content including video data and audio data basedon moving image coding standards such as HEVC (High Efficiency VideoCoding).

Predetermined transmission formats include, for example, MPEG-2 TS(Moving Picture Experts Group-2 Transport Stream) and MMT (MPEG MediaTransport) (see Non-Patent Literature 1). For example, Non-PatentLiterature 1 discloses a technique of transmitting encoded media dataper packet according to MMT.

There is an argument that a video image is hierarchized and encoded torealize scalability to decode and display encoded data, and atransmitting method for hierarchizing and transmitting encoded data isnot taken into account.

CITATION LIST Non-Patent Literature

-   NPL 1: Information technology—High efficiency coding and media    delivery in heterogeneous environment—Part1: MPEG media transport    (MMT), ISO/IEC DIS 23008-1

SUMMARY

In one general aspect, the techniques disclosed here feature atransmitting method for transmitting encoded data obtained byhierarchically encoding a video image into a basic layer and an extendedlayer, the transmitting method including: generating an encoded streamthat includes time information and the encoded data, the timeinformation indicating a time at which decoding or displaying processingof the encoded data is performed; and transmitting the generated encodedstream, and the encoded data includes a plurality of sets each includinga plurality of access units, each of the plurality of access units whichconfigure a first set can be independently decoded or can be decoded byreferring to decoded data of another access unit of the basic layer, thefirst set being a set for the basic layer among the plurality of sets,each of the plurality of access units which configure a second set canbe decoded by referring to decoded data of an access unit of the basiclayer, the second set being a set for the extended layer among theplurality of sets, and the time information includes first timeinformation which indicates a time at which the processing performed ona first access unit of the first set is performed and which is based ona reference clock, and second time information used to specify a time atwhich the processing performed on a second access unit of the second setis performed and which is based on the reference clock.

Additional benefits and advantages of the disclosed embodiments willbecome apparent from the specification and drawings. The benefits and/oradvantages may be individually obtained by the various embodiments andfeatures of the specification and drawings, which need not all beprovided in order to obtain one or more of such benefits and/oradvantages.

It should be noted that general or specific embodiments may beimplemented as a system, a method, an integrated circuit, a computerprogram, a storage medium, or any selective combination thereof.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a view illustrating an example of a picture predictedstructure in each layer hierarchically encoded to realize temporalscalability;

FIG. 2 is a view illustrating a relationship between a decoding time(DTS: Decoding Time Stamp) and a presentation time (PTS: PresentationTime Stamp) of each picture in FIG. 1;

FIG. 3 is a view illustrating a difference between DTSs of head picturesof each basic layer and an extended layer;

FIG. 4 is a view illustrating encoded data of each basic layer andencoded data of the extended layer;

FIG. 5 is a view for explaining a data structure of an encoded streamaccording to MMT;

FIG. 6 is a view for explaining a data structure of an encoded streamaccording to MMT;

FIG. 7 is a block diagram illustrating a configuration of a transmittingdevice according to an exemplary embodiment;

FIG. 8 is a flowchart of the transmitting method according to theexemplary embodiment;

FIG. 9 is a view illustrating an MP4 file including encoded data of thebasic layer and an MP4 file including encoded data of the extendedlayer;

FIG. 10 is a view illustrating a configuration example of random accessunits (RAU) of each basic layer and the extended layer;

FIG. 11 is a view illustrating an example where items of data of eachbasic layer and the extended layer are transmitted by using MMT;

FIG. 12 is a block diagram illustrating an example of a configuration ofa receiving device;

FIG. 13 is a view illustrating a flowchart illustrating an operation ofdetermining a DTS of an access unit included in the extended layer;

FIG. 14 is a view illustrating an example where the encoded data in FIG.1 is multiplexed according to MMT;

FIG. 15 is a view illustrating an example of a transmission packetsequence obtained when the items of encoded data of the basic layer andthe extended layer are converted into one encoded stream (packetsequence);

FIG. 16 is a block diagram illustrating another example of aconfiguration of the receiving device;

FIG. 17 is a flowchart indicating a receiving method for receiving theitems of encoded data of the basic layer and each extended layer;

FIG. 18 is a block diagram illustrating another example of aconfiguration of the receiving device; and

FIG. 19 is a flowchart of a receiving method.

DETAILED DESCRIPTION

(Base Knowledge of Present Disclosure)

Scalability to decode or display encoded data can be realized byhierarchically encoding access units of the encoded data. When, forexample, a frame rate obtained by decoding only encoded data of a lowerlayer among a plurality of items of encoded data encoded into aplurality of layers is 60 fps, a frame rate obtained by decoding encodeddata up to a higher layer becomes 120 fps.

In this regard, a case where encoded data of a lower layer and encodeddata of a higher layer are transmitted will be described. When areceiving device which has received encoded data of the lower layer andthe encoded data of the higher layer decodes the items of the encodeddata of the both layers to obtain a video image of a frame rate of 120fps, the receiving device needs to rearrange the received encoded dataof each layer in a decoding order. However, there are problems that itis necessary to rearrange the items of the received data in the decodingorder based on DTSs (Decoding Time Stamp: decoding time) or PTSs(Presentation Time Stamp: presentation time) of access units, and it isnot possible to uniquely determine the DTS or the PTS of each accessunit of an extended layer.

Coding methods such as MPEG-4 AVC and HEVC (High Efficiency VideoCoding) can realize scalability (temporal scalability) in a timedirection by using pictures B (bidirectional reference predictionpictures) to which a reference can be made from other pictures.

FIG. 1 is a view illustrating an example of a picture predictedstructure in each layer hierarchically encoded to realize temporalscalability.

In FIG. 1, TemporalId is a layer identifier of an encoded structure, andTemporalId indicates a deeper layer as a number of TemporalId becomeslarger. A plurality of square blocks indicates pictures, Ix in aplurality of blocks represents pictures I (intra-screen predictionpictures), Px indicates pictures P (forward reference predictionpictures), and Bx and bx indicate pictures B (bidirectional referenceprediction pictures). Further, x of Ix, Px and Bx indicates apresentation order. That is, x represents an order to display pictures.

Further, arrows between a plurality of pictures indicate referencerelationships, and, for example, each picture B4 indicates a predictionimage generated by using I0 and B8 as reference images.

In this regard, use of a picture having TemporalId larger thanTemporalId of this picture as a reference image is forbidden. Morespecifically, picture B2 whose TemporalId is 3 cannot use picture b1whose TemporalId is 4 as a reference image.

As illustrated in FIG. 1, a data structure of encoded data is defined asa plurality of layers to provide temporal scalability. When, forexample, all pictures whose TemporalIds are 0 to 4 in FIG. 1 aredecoded, a video image of 120 fps (frame per second) is obtained.Further, when only layers whose TemporalIds are 0 to 3 are decoded, avideo image of 60 fps is obtained. In FIG. 1, the layers whoseTemporalIds are 0 to 3 are basic layers, and the layer whose TemporalIDis 4 is an extended layer. That is, when only items of encoded data ofthe basic layers are decoded, a video image of 60 fps is obtained, and,when items of encoded data of the basic layers and the extended layerare decoded, a video image of 120 fps is obtained. In addition, this isan exemplary case, and the basic layers and the extended layer can beassociated in other combinations, too. Further, there may be layersother than two types of layers of the basic layers and the extendedlayer. That is, there may be three types of layers or more.

FIG. 2 is a view illustrating a relationship between a decoding time(DTS: Decode Time Stamp) and a presentation time stamp (PTS:Presentation Time Stamp) of each picture in FIG. 1. As illustrated inFIG. 2, a decoding order and a presentation order of a plurality ofpictures are different in some cases. In this case, picture I0 isdisplayed after picture B4 is decoded to prevent production of a gap indisplay processing. More specifically, it is indispensable that pictureb1 which comes immediately after picture I0 in the presentation orderhas been decoded to prevent production of a gap in the displayprocessing. Therefore, by displaying picture I0 after picture B4 isdecoded, it is possible to display picture b1 immediately after pictureI0 is displayed. That is, in this case, a time at which picture b1 canbe displayed immediately after picture I0 is displayed is set as apresentation time offset which is a time between a time at which pictureI0 is decoded and a time at which picture I0 is displayed.

In this regard, a layer to be decoded is switched according tocapability of receiving devices by allowing a receiving device which candecode items of encoded data up to the extended layer to reproduce avideo image of 120 fps, and allowing a receiving device which can decodeonly the items of encoded data of basic layers due to processingcapability to reproduce a video image of 60 fps. In this case, bytransmitting items of encoded data of the basic layers and encoded dataof the extended layer identifiably, each receiving device can decode theitems of received data by filtering the items of the received dataaccording to capability of each receiving device. That is, the receivingdevice which can decode the items of encoded data up to the extendedlayer can decode both of the items of encoded data of the basic layersand the encoded data of the extended layer, and the receiving devicewhich can decode only the encoded data of the basic layers can decodeonly the items of encoded data of the basic layers by filtering theitems of encoded data.

Further, each hierarchized encoded data is multiplexed in variousmultiplexing formats such as MPEG-2 TS (Transport Stream), MMT (MPEGMedia Transport), MPEG-DASH (Dynamic Adaptive Streaming over HTTP) orRTP (Real-time Transport Protocol), and is transmitted. In this regard,according to MMT and MPEG-DASH, multiplexing is basically performed byusing MP4 (a file format which is based on ISO Base Media File Format ofMPEG). According to MP4 in particular, information such as DTSs and PTSsis expressed as difference information of DTSs or PTSs between twocontinuous access units (corresponding to pictures in the case of avideo image).

FIG. 3 is a view illustrating a difference between DTSs of head picturesin each basic layer and the extended layer. More specifically, FIG. 3 isa view illustrating pictures arranged in the decoding order in FIG. 2 aspictures belonging to the basic layers and pictures belonging to theextended layer. FIG. 4 is a view illustrating encoded data of each basiclayer and encoded data of the extended layer.

As illustrated in FIG. 3, a difference (referred to as a “decoding timeoffset” below) is produced between DTS 11 of head picture I0 of eachbasic layer and DTS 21 of head picture b1 of the extended layer.However, an MP4-based multiplexing method has a problem that it ispossible to provide only relatively time information of each basic layeror the extended layer, and it is not possible to express a decoding timeoffset. That is, this multiplexing method has a problem that it is notpossible to specify a timing at which a picture of the extended layer isdecoded after the basic layers are decoded.

Hence, when items of encoded data of the basic layers and encoded dataof the extended layer are independently transmitted, the receivingdevice separately receives the items of the encoded data of the basiclayers and the encoded data of the extended layer as illustrated in FIG.4. In this case, when the items of encoded data of the both layers aredecoded, the items of encoded data of the both layers need to berearranged in the decoding order illustrated in FIG. 3, and input to adecoder (decoder). Hence, there is also a problem that processing ofobtaining a DTS per access unit and rearranging access units in thedecoding order based on the DTSs is performed, and a processing amountbefore decoding increases.

To solve these problems, a data transmitting method according to oneaspect of the present disclosure is a transmitting method fortransmitting encoded data obtained by hierarchically encoding a videoimage into a basic layer and an extended layer, and includes: generatingan encoded stream that includes time information and the encoded data,the time information indicating a time at which decoding or displayingprocessing of the encoded data is performed; and transmitting thegenerated encoded stream, and the encoded data includes a plurality ofsets each including a plurality of access units, each of the pluralityof access units which configure a first set can be independently decodedor can be decoded by referring to decoded data of another access unit ofthe basic layer, the first set being a set for the basic layer among theplurality of sets, each of the plurality of access units which configurea second set can be decoded by referring to decoded data of an accessunit of the basic layer, the second set being a set for the extendedlayer among the plurality of sets, and the time information includesfirst time information which indicates a time at which the processingperformed on a first access unit of the first set is performed and whichis based on a reference clock, and second time information used tospecify a time at which the processing performed on a second access unitof the second set is performed and which is based on the referenceclock.

According to this, even when the items of encoded data of the basiclayer and the extended layer are transmitted as items of different data,it is possible to specify a time at which the access unit of theextended layer is processed.

For example, the first access unit may be an access unit on which theprocessing is performed first, in the first set, and the second accessunit may be an access unit on which the processing is performed first,in the second set.

For example, the processing may be decoding, each of a plurality ofaccess units other than the first access unit of the first set may beassociated with a first relative time which is based on a time indicatedby the first time information, and a time at which each of a pluralityof access units of the first set is displayed may be specified based ona second relative time which is based on a time at which each of aplurality of access units is decoded, each of a plurality of accessunits other than the second access unit of the second set may beassociated with a third relative time which is based on a time indicatedby the second time information, and a time at which each of a pluralityof access units of the second set is displayed may be specified based ona fourth relative time which is based on a time at which each of aplurality of access units is decoded.

For example, the processing may be displaying, each of a plurality ofaccess units other than the first access unit of the first set may beassociated with a fifth relative time which is based on a time indicatedby the first time information, a time at which each of a plurality ofaccess units of the first set is displayed may be specified based on asixth relative time which is based on a time at which each of aplurality of access units is displayed, each of a plurality of accessunits other than the second access unit of the second set may beassociated with a seventh relative time which is based on a timeindicated by the second time information, and a time at which each of aplurality of access units of the second set is decoded may be specifiedbased on an eighth relative time which is based on a time at which eachof a plurality of access units is displayed.

For example, the second time information may be a time offset that is adifference value from a first absolute time indicated by the first timeinformation.

For example, the set may be a random access to which a random accessunit can be made, and the first access unit and the second access unitmay be random access points.

For example, the first set comprises a plurality of first sets and thesecond set comprises a plurality of second sets. Each of the pluralityof first sets is associated with a corresponding one of the plurality ofsecond sets.

For example, the second sets may be decoded by referring only to decodeddata of the first sets associated with the second sets on the one-on-onebasis.

For example, each of the second sets may further include second headerinformation in which identification information used to identify each ofthe first sets associated with the second sets on the one-on-one basis,and the second time information are stored.

For example, each of the first sets may include first header informationin which identification information used to identify each of the secondsets associated with the first sets on the one-on-one basis, and thesecond time information are stored.

For example, the encoded stream may further include associationinformation in which first identification information used to identifyeach of the first sets and second identification information used toidentify each of the second sets associated with each of the first setson the one-on-one basis are associated with each other.

For example, the time information may be stored in control informationof the encoded stream.

For example, the second time information may indicate a second absolutetime different from a first absolute time indicated by the first timeinformation.

For example, the generating step may include generating a first encodedstream including the first set and a second encoded stream including thesecond set, and the transmitting step may include transmitting the firstencoded stream by using a first channel, and transmitting the secondencoded stream by using a second channel different from the firstchannel.

For example, the generating step may include generating one of the firstencoded stream and the second encoded stream according to MPEG-2 TS(Moving Picture Experts Group-2 Transport Stream), and generating theother one of the first encoded stream and the second encoded streamaccording to MMT (MPEG Media Transport).

For example, one of the first channel and the second channel may be achannel for broadcasting, and the other one of the first channel and thesecond channel may be a channel used for communication.

In addition, these overall or specific aspects may be realized by a datareceiving method, an integrated circuit, a computer program or arecording medium such as a computer-readable CD-ROM, or may be realizedby an arbitrary combination of a data receiving method, the integratedcircuit, the computer program and the recording medium.

The data transmitting method and the data receiving method according toone aspect of the present disclosure will be specifically described withreference to the drawings.

In addition, each of the exemplary embodiments described below is onespecific example of the present disclosure. Numerical values, shapes,materials, components, arrangement positions and connection modes of thecomponents, steps and an order of the steps described in followingexemplary embodiments are exemplary, and do not intend to limit thepresent disclosure. Further, components which are not recited in anindependent claim which represents a top concept among the components inthe following exemplary embodiments will be described as arbitrarycomponents.

EXEMPLARY EMBODIMENTS

[Transmitting Method]

The transmitting method (transmitting device) according to the exemplaryembodiments will be describe below with reference to the drawings. Thetransmitting method for transmitting encoded data according to MMT willbe described as an example in the exemplary embodiments.

First, a data structure of an encoded stream according to MMT will bedescribed. FIGS. 5 and 6 are views for explaining data structures ofencoded streams according to MMT.

As illustrated in FIG. 5, encoded data includes a plurality of accessunits (AU). Encoded data is, for example, AV data encoded based onmoving image coding standards such as HEVC. More specifically, encodeddata includes video data, audio data, meta data accompanying the videodata and the audio data, still images, and files. When encoded data isvideo data, one AU is a unit corresponding to one picture (one frame).

According to MMT, encoded data is converted into MP4 data (an MP4 headeris given) according to an MP4 file format in GOP (Group Of Picture)units. That is, encoded data includes a plurality of sets (GOPs) eachincluding a plurality of access units. Each GOP is a random access pointof encoded data, and a head access unit of each GOP in a decoding ordercorresponds to an IDR (Instantaneous Decoding Refresh) picture accordingto HEVC or AVD or a picture I according to non-IDR. Each of a pluralityof these sets belongs to one of the basic layers and the extended layer.In this regard, a set belonging to the basic layers is the first set,and a set belonging to the extended layer is the second set.

In addition, each of a plurality of access units which configure thefirst set belongs to the basic layers and therefore can be independentlydecoded or can be decoded by referring to decoded data of another accessunit of each basic layer. In addition, each of a plurality of accessunits which configure the second set belongs to the extended layer andtherefore can be decoded by referring to decoded data of an access unitof each basic layer.

In the MP4 header included in the MP4 data, relative values of apresentation time (the above PTS) and a decoding time (the above DTS) ofan access unit are described. Further, in the MP4 header, sequencenumbers of the MP4 data are described. In addition, the MP4 data (MP4file) is an example of an MPU (Media Processing Unit) which is a dataunit defined according to the MMT standards. Only sample data in the MPUmay also be transmitted without transmitting an MP4 header in the MPU.In this case, too, the MPU corresponds to a random access unit, and asample which configures a MPU and the MPU are associated on a one-on-onebasis. Further, the MPU may include a plurality of GOPs.

Further, as illustrated in FIG. 6, encoded stream 10 according to MMTincludes control information 11, time information 12 and a plurality ofMMT packets 13. In other words, encoded stream 10 is a packet sequenceof MMT packets 13.

Encoded stream 10 (MMT stream) is one of one or more streams whichconfigure one MMT package. The MMT package corresponds to, for example,one broadcast program content.

Control information 11 includes information indicating that encodedstream 10 is a scalable-coded stream (a stream including both of basiclayers and an extended layer), and information of a type of scalablecoding and the number of layer levels (a number of layers). In thisregard, the type of scalable coding is temporal scalability, spatialscalability and SNR (Signal-to-Noise Ratio) scalability, and the numberof layer levels refers to the number of layers such as basic layers andthe extended layers.

Further, control information 11 includes, for example, informationindicating an association relationship between a plurality of assets andpacket IDs. In addition, each asset is a data entity including data ofsame transport characteristics, and is, for example, one of video dataand audio data.

Control information 11 is, more specifically, CI (CompositionInformation) and an MPT (MMT Package Table) according to MMT. Inaddition, control information 11 is a PMT (Program Map Table) accordingto MPEG2-TS, and is MPD (Media Presentation Description) according toMPEG-DASH.

Time information 12 is information used to determine a PTS or a DTS ofan access unit. Time information 12 is, more specifically, for example,a PTS or a DTS which is an absolute time of a head access unit of theMPU belonging to each basic layer. More specifically, time information12 can indicate an absolute value of the PTS of the head access unit ofthe MPU in the presentation order in the case of the PTS, and indicatean absolute time of the DTS of the head access unit of the MPU in thedecoding order in the case of the DTS. Further, time information 12 maybe stored as program information in control information 11. When timeinformation 12 is stored as the program information, for example, theprogram information is stored in an MMT message and time information 12can be stored as a descriptor in the program information.

When, for example, it is assumed that all of a plurality of pictures ofthe basic layers in FIG. 3 configure one first set, picture I0 that isthe first access unit to be decoded first in the decoding order of thefirst set is decoded at a time indicated by DTS 11. In this case, firsttime information indicating a time (DTS 11) at which the first accessunit of the first set is decoded and which is based on a reference clockmay be stored as time information 12 (first absolute time) of encodedstream 10. That is, the first absolute time indicates, for example, DTS11 itself.

Further, when it is assumed that all of a plurality of pictures of theextended layer in FIG. 3 configure one second set, picture b1 that isthe second access unit to be decoded first in the decoding order of thesecond set is decoded at a time indicated by DTS 21. In this case,second time information used to specify a time (DTS 21) at which thesecond access unit of the second set is decoded and which is based on areference clock is a decoding time offset that is a difference valuefrom the first absolute time indicated by the first time information asdescribed above. Similar to the first time information, the second timeinformation may be stored as time offset information (decoding timeoffset) of encoded stream 10. That is, DTS 21 is specified by adding thedecoding time offset indicated by the second time information, to DTS 11indicated by the first time information. Further, the absolute value ofthe time information of the access unit of the second set may be storedas the second time information instead of the time offset information ofthe first time information.

In addition, the reference clock is an NTP (Network Time Protocol) whenencoded streams are transmitted according to an MMT mode, and is a PCR(Program Clock Reference) when encoded streams are transmitted accordingto an MPEG2-TS mode. In this regard, the NTP needs to be a referenceclock set by the transmitting device and does not necessarily need tomatch with an NTP value of an NTP server typically used on the Internet.

MMT packet 13 is data obtained by converting MP4 data into a packet.According to the exemplary embodiments, one MMT packet 13 includes oneMP4 data (MPU). As illustrated in FIG. 6, MMT packet 13 includes header13 a (an MMT packet header or a TS packet header in the case ofMPEG2-TS) and payload 13 b.

In payload 13 b, MP4 data is stored. In addition, in payload 13 b,divided MP4 is stored in some cases.

Header 13 a is auxiliary information related to payload 13 b. Forexample, header 13 a includes a packet ID and time information. The timeinformation described herein is a relative value of a presentation time(PTS) or a decoding time (DTS) of MP4 data.

The packet ID is an identification number indicating an asset of dataincluded in MMT packet 13 (payload 13 b). The packet ID is a uniqueidentification number of each asset which configures an MMT package.

Thus, each encoded stream includes time information (the DTS or the PTS)indicating a time at which processing of decoding or displaying encodeddata is performed, and items of encoded data (ID1_#0, ID2_#0, ID1_#1,ID1_#2, ID2_#1, ID2_#2 and . . . in FIG. 6). This time informationincludes the above first time information and the second timeinformation.

FIG. 7 is a block diagram illustrating a configuration of a transmittingdevice according to the exemplary embodiments. FIG. 8 is a flowchart ofthe transmitting method according to the exemplary embodiments.

As illustrated in FIG. 7, transmitting device 15 includes encoder 16 andtransmitter 17. In addition, more specifically, components oftransmitting device 15 are realized by a microcomputer, a processor or adedicated circuit.

According to the transmitting method for transmitting encoded stream 10according to the exemplary embodiments, encoded stream 10 including timeinformation indicating a time at which a set including a plurality ofaccess units is decoded or displayed, and a plurality of access unitswhich configure the set is generated (S11: generating step).

Generated encoded stream 10 is transmitted by transmitter 17 by using achannel (S12: transmitting step).

First Exemplary Embodiment

Next, a transmitting method and a receiving method in a case whereencoded data of an extended layer is transmitted in an MP4-basedmultiplexing format will be more specifically described.

The MP4-based multiplexing format may be, for example, MMT, DASH or filedata of MP4. According to MMT, an MPU (Media Processing Unit)corresponds to an MP4 file, and, according to DASH, a segmentcorresponds to Movie Fragment of MP4.

As illustrated in FIG. 3, a multiplexing format which expresses timeinformation indicating a time at which each sample is decoded(displayed) as a relative time (a difference between samples) of DTSs(or PTSs) of a plurality of access units, and which does not indicate anabsolute value of time information indicating a time at which all of aplurality of access units of a set are decoded (displayed) is applicableto a format other than MP4, too. In addition, each sample describedherein refers to a unit at which data is handled in MP4 and correspondsto an access unit.

(Decoding Time and Presentation Time)

A decoding time (DTS) of an access unit of the extended layer will bedescribed first by using an MP4 file as an example. FIG. 9 is a viewillustrating an MP4 file (MP4b) including items of encoded data of thebasic layers and an MP4 file (MP4e) including encoded data of theextended layer.

As illustrated in FIG. 9, it is assumed that there is an MP4 file (MP4b)including the items of encoded data of the basic layers and an MP4 file(MP4e) including encoded data of the extended layer. In this regard,when a decoding time offset between the MP4b and the MP4e is dec_offset,a DTS of each sample of the extended layer in the MP4e is expressed asthe following equation.sample_e(i)_dec=sample_e(i)_dec_base+dec_offset  (Equation 1)

sample_e(i)_dec: a DTS of an ith sample of an extended layer

sample_e(i)_dec_base: a DTS of an uncorrected sample (a DTS of eachsample in a case where a DTS of a head sample is 0) calculated from adifference total sum of decoding times of 0th to ith samples of theextended layer (a total sum of sample_delta of ‘stts’ of MP4 orsample_duration of Movie Fragment).

According to the above (equation 1), a decoding time offset (dec_offset)is an offset value in a case where a DTS of a head sample of the basiclayer in a decoding order is assumed to be 0.

In this regard, when the DTS of the head sample of each basic layer isdelta, the DTS of each sample of the extended layer in the MP4e isexpressed by the following equation.sample_e(i)_dec=sample_e(i)_dec_base+dec_offset+delta  (Equation 2)

More specifically, in FIG. 3, DTS 23 of picture b5 that is a thirdsample of the extended layer can be calculated as follows by using(equation 2) since sample_e(i)_dec_base is calculated by adding relativetime 21 and relative time 22 and delta is DTS 11.DTS 23=relative time 21+relative time 22+dec_offset+DTS 11  (Equation 3)

Further, the DTS of each sample of the basic layer in the MP4b isexpressed by the following equation.sample_b(i)_dec=sample_b(i)_dec_base+dec_offset  (Equation 4)

sample_b(i)_dec: a DTS of an ith sample in each basic layer

sample_b(i)_dec_base: a DTS of an uncorrected sample (a DTS of eachsample in a case where a DTS of a head sample is 0) calculated from adifference total sum of decoding times of 0th to ith samples of thebasic layers (a total sum of sample_delta of ‘stts’ of MP4 orsample_duration of Movie Fragment).

In this regard, when the DTS of the head sample of each basic layer isdelta, the DTS of each sample of the extended layer in the MP4b isexpressed by the following equation.sample_b(i)_dec=sample_b(i)_dec_base+dec_offset+delta  (Equation 5)

That is, each of a plurality of access units other than a first accessunit of a first set is associated with first relative times (relativetimes 11 to 18) which are based on a time indicated by first timeinformation (e.g. DTS 11). Further, times (PTSs 11 to 19) at which eachof the plurality of access units of the first set is displayed isspecified based on a second relative time which is based on a time atwhich each of a plurality of access units is decoded. That is, forexample, the second relative time which is a difference from time DTS 13at which picture B8 is decoded is associated with picture B8, and timePTS 15 at which picture B8 is displayed is calculated by adding thesecond relative time associated with picture B8, to DTS 13.

Further, each of the plurality of access units other than a secondaccess unit of a second set is associated with third relative times(relative times 21 to 27) which are based on a time indicated by secondtime information (e.g. DTS 21). Furthermore, times (PTSs 21 to 28) atwhich each of the plurality of access units of the second set isdisplayed are specified based on a fourth relative time which is basedon a time at which each of the plurality of access units is decoded.That is, for example, the fourth relative time which is a differencefrom time DTS 26 at which picture b11 is decoded is associated withpicture b11, and time PTS 26 at which picture b11 is displayed iscalculated by adding the fourth relative time associated with pictureb11, to DTS 26.

(Storage of Decoding Time Offset)

In addition, there are the following three cases of storage destinationsof information (decoding time offset information) indicating a decodingtime offset.

(1) Case where decoding offset information is stored in MP4 fileincluding track of extended layer

The decoding time offset information includes at least a decoding timeoffset and identification information of a track of each basic layer.The identification information of the track of each basic layer includesa track ID of the track of each basic layer and identificationinformation of an MP4 file (a file name of an MP4 file) including thetrack of each basic layer. That is, the second set of the extended layerincludes second header information in which identification informationused to identify the first set associated with the second set, and thesecond time information are stored.

(2) Case where decoding offset information is stored in MP4 fileincluding track of basic layer

The decoding time offset information includes at least a decoding timeoffset and identification information of a track of the extended layer.That is, the first set of the basic layer includes first headerinformation in which identification information used to identify thesecond set associated with the first set, and the second timeinformation are stored.

(3) Case where decoding offset information is stored in information inwhich MP4 file including track of each basic layer and MP4 fileincluding track of extended layer are associated with each other

The decoding time offset information includes at least a decoding timeoffset, identification information of a track of each basic layer andidentification information of the track of the extended layer. That is,the encoded stream may include association information in which firstidentification information used to identify the first set and secondidentification information used to identify the second set associatedwith the first set on the one-on-one basis are associated with eachother, and the second time information may be stored in the associationinformation.

In addition, in the case of above (1) or (2), a Box for storing thedecoding time offset information is defined, and the decoding timeoffset information can be arranged right below the Box of a track levelor in the same level as or a higher layer of a track level. Further, thedecoding time offset information may be included by extending anexisting Box without defining a new Box.

Further, the decoding time offset may be realized by using a function ofan empty duration in ‘elst’ of ‘moov’ or ‘traf’ of ‘moo’. In this case,too, it is necessary to associate a track of the extended layer with atrack of each basic layer.

Further, in the case of above (3), the decoding time offset informationmay be stored in a track of an MP4 file independent from tracks of eachbasic layer and the extended layer or may be stored in different tracksin the same MP4 file.

When different tracks are stored in the same MP4 file, decoding timeoffset information can be stored in a higher Box than a Box in trackunits such as a level right below ‘moov’ or ‘moof’. In this case,identification information of an MP4 file is not necessary as decodingtime offset information.

When a DTS and a PTS are different in the case of MP4, while headerinformation of MP4 includes difference information of this difference(the second relative times or the fourth relative time), the differenceinformation is applied to the DTS in which the decoding time offset hasbeen reflected.

Values of time scales of tracks of each basic layer and the extendedlayer are desirably adjusted. When the time scales are different, in thecase of above (3), the time scale of the decoding time offsetinformation may be additionally indicated or use of the time scale ofthe track of one of each basic layer and the extended layer may bedefined in advance.

In addition, the decoding time offset is applied only to the DTS of theextended layer.

In addition, the MP4 file in which each basic layer or the extendedlayer is stored may include only the track of each layer or may includeother tracks.

Second Exemplary Embodiment

When MP4 data is reproduced while being received (progressive downloador HTTP streaming), a random access to a head of Movie Fragment can bemade to decode and reproduce the MP4 data.

As in Movie Fragment, units to which a random access can be made will bereferred to as random access units (RAU), and head data of each RAU willbe referred to as a random access point (RAP). That is, a set (GOP)including a plurality of access units will be referred to as a randomaccess unit, and first access units of the first set and second accessunits of the second set will be referred to as random access points. Inthis case, in the case of a RAU (i.e., the second set) including encodeddata of the extended layer, it is necessary to reflect a decoding timeoffset to determine a DTS of a sample of a RAU (corresponding to anaccess unit in data units of MP4).

FIG. 10 is a view illustrating a configuration example of RAUs of eachbasic layer and the extended layer. A RAUb in FIG. 10 indicates a RAU ofthe basic layer (first set), and a RAUe indicates a RAU of the extendedlayer (second set).

The RAUb and the RAUe are configured to form a pair. That is, each of aplurality of first sets is associated with a corresponding one of aplurality of second sets. Samples that configure one RAUe refer to asample included in a corresponding RAUb which forms a pair with theRAUe, but do not refer to samples included in other RAUbs. That is, thesecond set can be decoded by referring only to decoded data of the firstset associated with the second set on the one-on-one basis. Hence, it isguaranteed that, by obtaining the RAUb and the RAUe which form a pair,it is possible to decode the samples included in the RAUs of both ofeach basic layer and the extended layer.

In this regard, the RAUs which form a pair can be associated with eachother based on sequence numbers used to identify the RAUs. In this case,a decoding time of a head sample of the RAU of the extended layer in thedecoding order can be determined by adding a decoding time offset to adecoding time of a head sample of the pairing RAU of each basic layer inthe decoding order.

A DTS that becomes a head sample of the RAU of each basic layer in thedecoding order is a first absolute time. The first absolute time is, forexample, a time defined according to UTC (Coordinated Universal Time).The first absolute time may be stored in content management informationsuch as a PMT (Program Map Table) according to MPEG-2 TS or in contentmanagement information obtained prior to reception of content.Alternatively, information indicating an absolute value of a DTS may bestored in header information of Movie Fragment. In addition, a decodingtime of the head sample of the RAU of the basic layer may be arbitrarilyset by the receiving device.

The decoding time offset information is necessary only for a first RAUto which a random access has been made. Therefore, by reflecting thedecoding time offset in the head sample of the first RAU in the decodingorder, decoding times of subsequent samples of the first RAU and samplesof RAUs subsequent to the first RAU can be determined by successivelyadding difference information of DTSs of continuous samples included inheader information of MP4.

Consequently, information indicating that the decoding time offsetinformation is necessary only when the DTS of the head sample of thefirst RAU after the random access is determined may be additionallystored.

Information indicating whether or not the RAUs of each basic layer andthe extended layer form a pair may be indicated in an MP4 file includingthe track of each basic layer or the extended layer or in contentmanagement information.

In addition, the RAUs of each basic layer and the extended layer may notnecessarily form a pair. When a pair is not formed, informationindicating an absolute value of the DTS of the head sample of the RAU ofthe extended layer in the decoding order may be indicated in headerinformation or content management information of an MP4 file. Inaddition, even when the RAUs of each basic layer and the extended layerform a pair, information indicating the absolute value of the DTS of thehead sample of the RAU of the extended layer may be stored. That is, thesecond time information may indicate a second absolute time differentfrom a first absolute time indicated by the first time information.

Further, when, for example, each basic layer is transmitted by usingMPEG-2 TS and the extended layer is transmitted by using DASH or MMT,the RAUs cannot be paired unless a signaling method for the RAUs isadditionally defined according to TS. In such a case, the decoding timeoffset of the extended layer or the absolute value of the DTS of thehead sample in the RAU can be desirably obtained from the contentmanagement information.

In addition, in the case of MPEG-2 TS, too, it is possible to indicate aboundary between RAUs based on header information in a TS packet or aPES packet in which signaling information of the RAU is stored. Further,when transmission is performed by using a format for streaming such asRTP, boundary information such as a sequence number of a RAU can beindicated in a payload header of a RTP packet. When RTP is used, it ispossible to describe identification information of sessions of eachbasic layer and the extended layer and a dependence relationship in metainformation for describing sessions such as a SDP (Session DescriptionProtocol).

Upon decoding, the samples of each basic layer and the extended layerare rearranged in the decoding order based on DTSs of the RAU of eachbasic layer and the RAU of the extended layer, and are input to adecoder. In this regard, when items of encoded data of each basic layerand the extended layer are arranged in the decoding order in receiveddata, samples do not need to be rearranged.

Third Exemplary Embodiment

FIG. 11 is a view illustrating an example where items of data in eachbasic layer and an extended layer are transmitted by using MMT.

FIG. 11 illustrates that each basic layer and the extended layer aretransmitted as different assets, and an MPU corresponds to a randomaccess unit. FIG. 11 illustrates an MPU of the basic layer as an MPUb,and an MPU of the extended layer as an MPUe. When the MPUb of each basiclayer and the MPUe of the extended layer form a pair, a DTS of a headsample of the MPUe of the extended layer can be determined in the sameway as a RAU of MP4 data described with reference to FIG. 10.

In this regard, the MPU corresponds to an MP4 file instead of MovieFragment. Therefore, decoding time offset information may be storedright below ‘moov’ or right below ‘mmpu’ which is a Box indicatingattribute information of the MPU such as an MPU sequence number.

Identification information of tracks of each basic layer and theextended layer of decoding time offset information may be onlyinformation (a file name, an asset ID or an MPU sequence number) used toidentify an MP4 file indicating an MPU when the MPU includes only onetrack including encoded data of each asset.

When DASH is used, too, a segment (more accurately, Media segment)corresponds to one or more Movie Fragments. Consequently, it is possibleto determine a decoding time in the same way as the above methodperformed for MP4 data.

According to DASH, although TS data can also configure a segment, MP4(ISO Base Media File Format) configures a segment.

[Receiving Method]

FIG. 12 is a block diagram illustrating an example of a configuration ofa receiving device. FIG. 13 is a view illustrating a flowchartillustrating an operation of determining a DTS of an access unitincluded in the extended layer.

An example of the operation of determining a DTS of an access unit whichconfigure the extended layer to decode the access unit of the extendedlayer will be described.

As illustrated in FIG. 12, receiving device 20 includes obtaining unit21, start deciding unit 22, offset reflector 23 and subsequent DTSdeterminator 24. In addition, more specifically, components of thereceiving device are realized by a microcomputer, a processor or adedicated circuit.

In this regard, receiving device 20 determines whether to decode onlyeach basic layer or decode both of each basic layer and the extendedlayer, in a previous step of step S21 according to information selectedby a user in advance or decoding capability of the receiving device.

First, obtaining unit 21 of receiving device 20 receives encoded stream10, analyzes decoding time offset information of encoded stream 10 andobtains a track of the basic layer, the track of the extended layer anda decoding time offset (S21).

In addition, when the decoding time offset information is included in afile or in a track in which encoded data of the extended layer isstored, processing in step S21 may be performed between next step S22and step S23.

Further, when information indicating an absolute value of a DTS of ahead access unit of a RAU of the extended layer in a decoding order isindicated, the DTS of the access unit of the extended layer can beobtained only from information of the extended layer. In this regard, aDTS of an access unit of the basic layer which starts being decoded isused to determine an access unit which starts being decoded.

Next, start deciding unit 22 of receiving device 20 decides whether ornot processing is processing of the access unit which starts beingdecoded (S22).

When start deciding unit 22 of receiving device 20 decides that theprocessing is the processing of the access unit which starts beingdecoded (Yes in S22), offset reflector 23 determines an access unit ofthe extended layer which starts being decoded first and calculates a DTSwhich reflects a decoding time offset (S23). More specifically, byadding the decoding time offset to a first absolute time which is theDTS of the access unit of the basic layer which starts being decodedfirst, the DTS of the access unit of the extended layer which startsbeing decoded first is calculated.

In this regard, when the access unit of the basic layer which startsbeing decoded is AU_b, the access unit of the extended layer whose DTScomes immediately after a DTS of Au_b is the access unit (AU_e) of theextended layer which starts being decoded.

When RAUs of the basic layer and the extended layer form a pair, the RAUof the extended layer which forms a pair with the RAU of each basiclayer which starts being decoded is a RAU of the extended layer whichstarts being decoded. The head access unit of the RAU which starts beingdecoded in the decoding order is AU_e. The RAU of the extended layerwhich forms a pair with the RAU of each basic layer can be obtained bysearching for an MPU whose MPU sequence number is the same as an MPUsequence number of each basic layer. In this regard, the MPU sequencenumber can be stored in header information of an MMT packet.

When the RAUs of each basic layer and the extended layer do not form apair, the access unit of the extended layer whose DTS comes immediatelyafter AU_b is searched and the access unit is decided as AU_e. That is,it is possible to determine the DTS based on information indicating anabsolute value of the DTS of a head sample of the RAU of the extendedlayer in the decoding order.

In addition, a DTS is determined according to whether the RAUs of thebasic layer and the extended layer form a pair or do not form a pair.However, the above operation may be switched according to theinformation as long as the information indicates whether or not the bothlayers form a pair.

Further, when a decoding time of an access unit at a file head or a RAUhead of the basic layer is not 0 (delta is not 0 and, for example, anabsolute time such as start at 17:00:00 is indicated), a value of deltais additionally added to determine a DTS. Alternatively, a valueobtained by adding the decoding time offset and delta may be indicatedas decoding time offset information.

When the number of layers is three or more (there are two types of theextended layers or more), information used to identify the differentextended layers from each other may be stored to select and determinethe extended layer to decode upon reproduction. In this case, thedecoding time offset information is set per different extended layer.

When start deciding unit 22 of receiving device 20 decides thatprocessing is not the processing of the access unit which starts beingdecoded (No in S22), subsequent DTS determinator 24 of receiving device20 adds a difference (relative time) between DTSs of a current accessunit and a last access unit, to the DTS of the last access unit in thedecoding order (S24).

In addition, offset reflector 23 and subsequent DTS determinator 24obtain header information of MP4 (e.g. ‘stbl’ of ‘trak’ or ‘trun’ of‘traf’) which is necessary to calculate the DTS based on an input whichis not illustrated.

(RAU of Extended Layer)

In this regard, definitions of each RAP and each RAU of each extendedlayer will be described in detail.

Each extended layer is decoded by referring to a decoding result of thebasic layer, and therefore it is not possible to decode each extendedlayer alone. Hence, when each extended layer alone is taken intoaccount, there is no RAP. However, to efficiently search for an accessunit of the extended layer corresponding to a RAP of the basic layerupon an random access to content, a RAP and a RAU are desirably definedfor each extend layer, too.

The RAU of each extended layer can be defined as follow. In addition,the RAP is a head access unit of the RAU in the decoding order. That is,a RAUe of each extended layer is a unit which forms a pair with a RAUbof the basic layer. Further, the RAUe of the extended layer is a unitused as the RAP of the basic layer such as Movie Fragment according toMP4, an MPU according to MMT or a segment according to DASH.

Furthermore, the head access unit of the RAU of the extended layer inthe decoding order may not be an access unit (sync sample according toMP4) such as a picture I or a picture IDR which can be independentlydecoded. Still further, information indicating sync sample may not beset.

Moreover, upon reproduction in the receiving device, a reference to RAUinformation of each extended layer is made to search for the access unitof each extended layer corresponding to the access unit of the basiclayer which starts being decoded. For example, a DTS of a head accessunit per RAU is searched for.

When searching for the RAU of each extended layer, the receiving devicemay perform an operation assuming that Movie Fragment or an MPUcorresponds to the RAU. Information indicating a unit of the RAU of eachextended layer may be stored in a track of each extended layer or an MP4file including the track of each extended layer. When the RAU of eachextended layer is searched for, whether or not the head access unit ofthe RAU is sync sample is ignored. Alternatively, without storinginformation of each extended layer for making a random access, after arandom access point of the basic layer is determined, an operation maybe performed to start decoding an MPU of each extended layer having thesame sequence number as a sequence number of the MPU of the basic layerwhich is the random access point.

Alternatively, each extended layer may be regarded such that the RAP ofeach extended layer corresponds to sync sample. In this case, in a tableof access units such as ‘stss’ or ‘mfra’ of MP4 data to which a randomaccess can be made, the RAP of each extended layer may be indicated. Thereceiving device can search for each RAP based on these tables. Further,according to Movie Fragment, flag information indicating whether or nota head sample is sync sample may be set in ‘traf’, and the receivingdevice may search for each RAP according to whether or not a sample issync sample.

The above applies likewise even when a RAU is defined according toMPEG-2 TS.

(Others)

That MP4 data includes an access unit of each extended layer may besignaled in a higher layer than a layer of MP4 data such as data in MP4,a PMT according to MPEG-2 TS or content management information. The datain MP4 is, for example, a brand of an MP4 file or ‘mmpu’ in the case ofan MPU according to MMT.

In content distribution which uses broadcast and communication incombination, the basic layer may be transmitted by way of broadcastingand each extended layer may be transmitted through a communicationnetwork. That is, the generating step of the transmitting method mayinclude generating a first encoded stream including the basic layer(first set) according to MPEG-2 TS, and generating a second encodedstream including each extended layer (second set) according to MMT.Further, a transmitting step may include transmitting the first encodedstream by using a channel used for broadcasting, and transmitting thesecond encoded stream by using a channel used for communication. Inaddition, the first encoded stream and the second encoded stream may begenerated according to a method opposite to the above method. Further,in this case, the first encoded stream and the second encoded stream aretransmitted by using channels opposite to the channels used in thetransmitting step.

Further, in a PMT (Program Map Table) for broadcasting, informationwhich is necessary to obtain encoded data of each extended layer such asa URL of a transmission source server of encoded data of the extendedlayer, an access destination file name or an access method (HTTP GET inthe case of download or a command of RTSP (Real Time Streaming Protocol)in the case of streaming). A method for storing, in the PMT, informationindicating a data access destination which is configuration data of thesame content and is transmitted from a different channel from thechannel for broadcast, and an access method is not limited to theexample of each basic layer and the extended layer, and is widelyapplicable to a combination of video data and audio data.

(Method for Transmitting Items of Encoded Data of Basic Layer andExtended Layer in Decoding Order)

When items of encoded data of the basic layer and each extended layerare transmitted as one encoded stream, it is possible to transmit theitems of encoded data of both of the layers in the decoding order. Inaddition, this method is applicable not only to transmission but also tostorage.

FIG. 14 is a view illustrating an example where the encoded data in FIG.1 is multiplexed according to MMT. The basic layer is stored in asset 1and each extended layer is stored in asset 2, and asset 1 includes anMPU_b and asset 2 includes an MPU_e.

According to MMT, an MPU of each asset is converted into a packet suchas an MMT packet or an RTP (Ream-time Transport Protocol) packet and istransmitted. In this case, the MPU is converted into a packet to arrangethe items of encoded data of the basic layer and the extended layerstored in a payload of the packet in the decoding order. A center linein FIG. 14 indicates an order of items of data stored in the payload ofthe packet when the MPU_b and the MPU_e are converted into a packet andare transmitted, and matches with the decoding order of the items ofencoded data.

Thus, by arranging the items of encoded data of the both layers in thedecoding order to transmit the items of encoded data of the basic layerand each extended layer as one encoded stream, the receiving device canobtain data of a corresponding access unit of the extended layer bystarting decoding RAPs from a RAP of the basic layer in order.Consequently, it is not necessary to perform an operation of rearrangingitems of encoded data of the basic layer and each extended layer in thedecoding order, and reduce a processing load of the receiving device.

According to a coding method such as MPEG-4 AVC or HEVC, the decodingorder and the presentation order can be obtained from items of encodeddata. Consequently, in the case of a fixed frame rate, it is possible todetermine a DTS and a PTS of the access unit of each extended layerbased on a DTS, a PTS and a frame rate of the basic layer. In this case,decoding time offset information of each extended layer may not besignaled

(Data Receiving Method for Receiving Encoded Data of Basic Layer andExtended Layer in Decoding Order)

A receiving method for receiving streams transmitted by the transmittingmethod described with reference to FIG. 14 will be described.

FIG. 15 is a view illustrating an example of a transmission packetsequence obtained when the items of encoded data of the basic layer andeach extended layer are converted into one encoded stream (packetsequence). As illustrated in FIG. 15, MPUs of the basic layer and eachextended layer which are encoded streams to be transmitted (encodedstreams received by the receiving device) are converted into packets ina transmission packet sequence. A packet in which the encoded data ofthe basic layer is stored and a packet in which the encoded data of theextended layer is stored are distinguished based on identificationinformation stored in a packet header. For the identificationinformation, packet_id can be used in the case of an MMT packet, PID canbe used in the case of a TS packet and SSID can be used in the case ofan RTP packet. When an MPU is converted into a packet according to amethod other than the MMT packet, for example, it is possible to definea TS packet sequence as an MPU, and store the MPU in a TS payload.Further, the payload may not be the MPU, and, in the case of the TSpacket, a packet obtained by converting encoded data into a PES packetmay be stored in a TS payload.

Further, the both layers do not need to be transmitted in the samepacket format, and may be converted into packets in different formatssuch as the TS packet for the basic layer and the MMT packet for eachextended layer.

Further, according to DASH, segments of the basic layer and eachextended layer may be classified, and items of data of access units ofboth segments may be stored in the decoding order.

The number of layers may be three or more (e.g. the basic layer and thetwo extended layers), and, even in this case, items of encoded data ofall layers are transmitted in the decoding order.

FIG. 16 is a block diagram illustrating another example of aconfiguration of the receiving device according. FIG. 17 is a flowchartindicating a receiving method for receiving the items of encoded data ofthe basic layer and each extended layer.

As illustrated in FIG. 16, receiving device 30 includes decode startposition determinator 31, decoding mode selector 32, data obtaining unit33 and decoder 34. In addition, more specifically, components ofreceiving device 30 are realized by a microcomputer, a processor or adedicated circuit.

First, decode start position determinator 31 of receiving device 30obtains a packet in which encoded data of the basic layer is stored, anddetermines an access unit of the basic layer which starts being decoded(S31). In this case, decode start position determinator 31 obtains atleast the packet of the basic layer based on additionally obtainedidentification information of the packet and determines the access unitwhich starts being decoded based on a random access point of the basiclayer.

Next, decoding mode selector 32 of receiving device 30 decides whetheror not to decode encoded data of each extended layer (S32).

When decoding mode selector 32 decides to decode the encoded data ofeach extended layer (Yes in S32), data obtaining unit 33 obtains both ofthe packet in which the encoded data of the basic layer is stored andthe packet in which the encoded data of each extended layer is stored(S33: mode 2). In this regard, when, for example, a packet ID of thebasic layer is 1 and a packet ID of each extended layer is 2, both ofthe packets whose packet IDs are 1 an 2 may be obtained. Alternatively,each of the ID packets may be separately filtered, and, when the packetID is 2, a step of regarding the packet ID as 1, and handling the packetID in the same way as the packet ID which is 1 may be provided. That is,in this case, only packets whose IDs are 1 are obtained.

In addition, identification information indicating whether or not thedecoding order is an ascending order may be provided to controlinformation (a PA table or an MP table) according to MMT or controlinformation in the PMT according to MPEG-2 TS. Receiving device 30 mayanalyze the identification information, and may not perform processingof rearranging items of encoded data of the basic layer and eachextended layer when the decoding order is the ascending order and mayperform the rearrangement processing when the decoding order is not theascending order.

When decoding mode selector 32 decides not to decode that the encodeddata of each extended layer (Yes in S32), data obtaining unit 33 obtainsonly the packet in which the encoded data of the basic layer is stored(S34: mode 1).

Further, decoder 34 of receiving device 30 decodes access units obtainedaccording to mode 1 and mode 2 in order (S35). In addition, even in thecase of mode 2, the items of encoded data of both of the basic layer andeach extended layer are arranged in the decoding order, and do not needto be rearranged. Further, decoded data is, for example, data of 60 fpsin the case of mode 1 and is data of 120 fps in the case of mode 2, andis displayed (reproduced) according to scalability of each extendedlayer.

Modified Example 1

In addition, the most simplified receiving device and receiving methodwill be described.

FIG. 18 is a block diagram illustrating another example of aconfiguration of the receiving device according. FIG. 19 is a viewillustrating a flowchart of a receiving method.

As illustrated in FIG. 18, receiving device 40 includes receiver 41 anddecoder 42. In addition, more specifically, components of the receivingdevice are realized by a microcomputer, a processor or a dedicatedcircuit.

First, receiver 41 of receiving device 40 receives time informationindicating a time at which processing of decoding encoded data isperformed, and an encoded stream including the encoded data (S41).

Next, decoder 42 of receiving device 40 decodes the encoded data of thereceived encoded stream at a time indicated by the time information(S42).

In addition, the time information may indicate a time at which displayprocessing is performed. In this case, a display which is notillustrated performs processing of displaying the data decoded bydecoder 42 at a time indicated by the time information.

Modified Example 2

Further, an example of scalability in a time direction has beendescribed above. However, the same method is applicable to scalabilityin a spatial direction when DTSs of a basic layer and an extended layerwhich configure an identical frame are different.

Modified Example 3

Further, an absolute value of a DTS of a head sample of the basic layerhas been indicated in each of the above exemplary embodiments. However,the present disclosure is not limited to this, and an absolute value ofa DTS may be determined by indicating an absolute value of a PTS of ahead sample of the basic layer in a presentation order and obtaining adifference between the PTS and the DTS from header information of MP4.Further, the difference between the DTS and the PTS (i.e., a secondrelative time and a fourth relative time) is stored in the headerinformation of MP4. Hence, instead of the absolute value of the DTS, theabsolute value of the PTS may be indicated.

That is, in this case, first time information is information indicatinga time at which a first access unit of a first set is displayed, andwhich is based on a reference clock. Further, second time information isinformation used to specify a time at which a second access unit of asecond set is displayed, and which is based on a reference clock.Furthermore, each of a plurality of access units other than the firstaccess unit of the first set may be associated with a fifth relativetime which is based on a time indicated by the first time information. Atime at which each of a plurality of access units of the first set isdecoded is specified based on a sixth relative time which is based on atime at which each of a plurality of access units is displayed. Further,each of a plurality of access units other than the second access unit ofthe second set is associated with a seventh relative time which is basedon a time indicated by the second time information. Furthermore, a timeat which each of a plurality of access units of the second set isdecoded is specified based on an eighth relative time which is based ona time at which each of a plurality of access units is displayed.

In addition, in each of the above exemplary embodiments, each componentmay be realized by being configured by dedicated hardware or executing asoftware program suitable to each component. Each component may berealized by causing a program executing unit such as a CPU or aprocessor to read and execute a software program recorded in a recordingmedium such as a hard disk or a semiconductor memory. In this regard,the software which realizes the transmitting device and the receivingdevice according to each of the above exemplary embodiments is afollowing program.

Further, this program is a transmitting method for causing a computer totransmit encoded data obtained by hierarchically encoding a video imageinto a basic layer and an extended layer, and includes: a generatingstep of generating an encoded stream including time information whichindicates a time at which processing of decoding or displaying theencoded data is performed, and the encoded data; and a transmitting stepof transmitting the generated encoded stream, and the encoded dataincludes a plurality of sets each including a plurality of access units,each of a plurality of access units which configure a first set that isthe set of the basic layer among a plurality of sets can beindependently decoded or can be decoded by referring to decoded data ofanother access unit of the basic layer, each of a plurality of accessunits which configure a second set that is the set of the extended layeramong a plurality of sets can be decoded by referring to decoded data ofthe access unit of the basic layer, and the time information includesfirst time information which indicates a time at which the processingperformed on a first access unit of the first set is performed and whichis based on a reference clock, and second time information used tospecify a time at which the processing performed on a second access unitof the second set is performed and which is based on the referenceclock.

Further, this program is a receiving method for causing a computer toreceive encoded data obtained by hierarchically encoding a video imageinto a basic layer and an extended layer, and includes: a receiving stepof receiving an encoded stream including time information whichindicates a time at which processing of decoding or displaying theencoded data is performed, and the encoded data; and a processing stepof performing the processing on the encoded data of the received encodedstream at a time indicated by the time information, and the encoded dataincludes a plurality of sets each including a plurality of access units,each of a plurality of access units which configure a first set that isthe set of the basic layer among a plurality of sets can beindependently decoded or can be decoded by referring to decoded data ofanother access unit of the basic layer, each of a plurality of accessunits which configure a second set that is the set of the extended layeramong a plurality of sets can be decoded by referring to decoded data ofthe access unit of the basic layer, and the time information includesfirst time information which indicates a time at which the processingperformed on a first access unit of the first set is performed, andsecond time information used to specify a time at which the processingperformed on a second access unit of the second set is performed.

Further, in the above exemplary embodiments, another processor mayexecute processing executed by a specific processor. Furthermore, anorder of a plurality of processings may be changed, or a plurality ofprocessings may be executed in parallel.

In addition, comprehensive or specific aspects of the present disclosuremay be realized by a system, a method, an integrated circuit, a computerprogram or a recording medium such as a computer-readable CD-ROM.Further, comprehensive or specific aspects of the present disclosure maybe realized by an arbitrary combination of the system, the method, theintegrated circuit, the computer program and the recording medium.

The transmitting method and the receiving method according to one or aplurality of aspects of the present disclosure have been describedabove. However, the present disclosure is not limited to these exemplaryembodiments. Aspects variously modified by one of ordinary skill in theart or aspects realized by combining components of different exemplaryembodiments may also be included in one or a plurality of aspects of thepresent disclosure without departing from the spirit of the presentdisclosure.

The present disclosure is useful for the transmitting method and thereceiving method which can specify a processing time of an access unitof the extended layer.

What is claimed is:
 1. A transmitting method for transmitting encoded data obtained by hierarchically encoding a video image into a basic layer and an extended layer, the transmitting method comprising: generating an encoded stream that includes time information and the encoded data, the time information indicating a time at which decoding or displaying processing of the encoded data is performed; and transmitting the generated encoded stream, wherein the encoded data includes a plurality of sets each including a plurality of access units, each of the plurality of access units which configure at least one first set is independently decoded or is decoded by referring to decoded data of another access unit of the basic layer, the at least one first set being for a set of the basic layer among the plurality of sets, each of the plurality of access units which configure at least one second set is decoded by referring to decoded data of an access unit of the basic layer, the at least one second set being a set for the extended layer among the plurality of sets the time information includes first time information which indicates a time at which the processing performed on a first access unit of the first set is performed and which is based on a reference clock, and second time information used to specify a time at which the processing performed on a second access unit of the second set is performed and which is based on the reference clock, the plurality of access units which configure the at least one second set is displayed at a frame rate higher than a frame rate for displaying only the plurality of access units which configure the at least one first set among the plurality of access units which configure the at least one first set and the plurality of access units which configure the at least one second set, by being displayed alternately with the plurality of access units which configure the at least one first set, and each of the plurality of access units which configure the at least one first set is decoded according to a first relative time based on the first time information by referring to the decoded data of the another access unit of the basic layer, the first relative time being a difference in decoding or displaying times of the access units.
 2. The transmitting method according to claim 1, wherein the first access unit is an access unit of the first set on which the processing is performed first, and the second access unit is an access unit of the second set on which the processing is performed first.
 3. The transmitting method according to claim 2, wherein the processing is decoding, each of the plurality of access units other than the first access unit of the first set is associated with the first relative time which is based on a time indicated by the first time information, a time at which each of the plurality of access units of the first set is displayed is specified based on a second relative time which is based on a time at which each of the plurality of access units is decoded, each of the plurality of access units other than the second access unit of the second set is associated with a third relative time which is based on a time indicated by the second time information, and a time at which each of the plurality of access units of the second set is displayed is specified based on a fourth relative time which is based on a time at which each of the plurality of access units is decoded.
 4. The transmitting method according to claim 2, wherein the processing is displaying, each of the plurality of access units other than the first access unit of the first set is associated with a fifth relative time which is based on a time indicated by the first time information, a time at which each of the plurality of access units of the first set is decoded is specified based on a sixth relative time which is based on a time at which each of the plurality of access units is displayed, each of the plurality of access units other than the second access unit of the second set is associated with a seventh relative time which is based on a time indicated by the second time information, and a time at which each of the plurality of access units of the second set is decoded is specified based on an eighth relative time which is based on a time at which each of the plurality of access units is displayed.
 5. The transmitting method according to claim 1, wherein the second time information is a time offset that is a difference value from a first absolute time indicated by the first time information.
 6. The transmitting method according to claim 1, wherein the first set is a random access unit to which a random access is made, and the first access unit and the second access unit are random access points.
 7. The transmitting method according to claim 6, wherein each of the at least one first set is associated with a corresponding one of the at least one second set.
 8. The transmitting method according to claim 7, wherein the second sets is decoded by referring only to decoded data of the first sets associated with the second sets on the one-on-one basis.
 9. The transmitting method according to claim 8, wherein each of the second sets further includes second header information in which identification information used to identify each of the first sets associated with the second sets on the one-on-one basis, and the second time information are stored.
 10. The transmitting method according to claim 8, wherein each of the first sets further includes first header information in which identification information used to identify each of the second sets associated with the first sets on the one-on-one basis, and the second time information are stored.
 11. The transmitting method according to claim 8, wherein the encoded stream includes association information in which first identification information used to identify each of the first sets and second identification information used to identify each of the second sets associated with each of the first sets on the one-on-one basis are associated with each other, and the second time information is stored in the association information.
 12. The transmitting method according to claim 1, wherein the time information is stored in control information of the encoded stream.
 13. The transmitting method according to claim 1, wherein the second time information indicates a second absolute time different from a first absolute time indicated by the first time information.
 14. The transmitting method according to claim 1, wherein the generating step includes generating a first encoded stream including the first set and a second encoded stream including the second set, and the transmitting step includes transmitting the first encoded stream by using a first channel, and transmitting the second encoded stream by using a second channel different from the first channel.
 15. The transmitting method according to claim 14, wherein the generating includes generating one of the first encoded stream and the second encoded stream according to MPEG-2 TS (Moving Picture Experts Group-2 Transport Stream), and generating the other one of the first encoded stream and the second encoded stream according to MMT (MPEG Media Transport).
 16. The transmitting method according to claim 14, wherein one of the first channel and the second channel is a channel for broadcasting, and the other one of the first channel and the second channel is a channel used for communication.
 17. The transmitting method according to claim 1, wherein the first time information includes a first display time at which a first basic access unit is displayed, and a second display time at which a second basic access unit to be displayed subsequently to the first basic access unit is displayed, the first basic access unit and the second basic access unit being included in the plurality of access units which configure the at least one first set, and the second time information includes a third display time at which an extended access unit is displayed among extended access units which configure the at least one second set, the third display time being a time between the first display time and the second display time.
 18. The transmitting method according to claim 1, wherein each of the plurality of access units which configure the at least one second set is decoded according to a second relative time based on the second time information by referring to the decoded data of the access unit of the basic layer.
 19. The transmitting method according to claim 1, wherein the second time information used to specify the time at which the processing performed on the second access unit of the second set is performed is an absolute value or a difference value from the time indicated by the first time information.
 20. A receiving method for receiving encoded data obtained by hierarchically encoding a video image into a basic layer and an extended layer, the receiving method comprising: receiving an encoded stream including time information which indicates a time at which processing of decoding or displaying the encoded data is performed, and the encoded data; and performing the processing on the encoded data of the received encoded stream at a time indicated by the time information, wherein the encoded data includes a plurality of sets each including a plurality of access units, each of the plurality of access units which configure a first set that is the set of the basic layer among the plurality of sets is independently decoded or is decoded by referring to decoded data of another access unit of the basic layer, each of the plurality of access units which configure a second set that is the set of the extended layer among the plurality of sets is decoded by referring to decoded data of the access unit of the basic layer, and the time information includes first time information which indicates a time at which the processing performed on a first access unit of the first set is performed, and second time information used to specify a time at which the processing performed on a second access unit of the second set is performed, the plurality of access units which configure the at least one second set is displayed at a frame rate higher than a frame rate for displaying only the plurality of access units which configure the at least one first set among the plurality of access units which configure the at least one first set and the plurality of access units which configure the at least one second set, by being displayed alternately with the plurality of access units which configure the at least one first set, and each of the plurality of access units which configure the at least one first set is decoded according to a first relative time based on the first time information by referring to the decoded data of the another access unit of the basic layer, the first relative time being a difference in decoding or displaying times of the access units.
 21. The receiving method according to claim 20, wherein each of the plurality of access units which configure the at least one second set is decoded according to a second relative time based on the second time information by referring to the decoded data of the access unit of the basic layer.
 22. The receiving method according to claim 20, wherein the second time information used to specify the time at which the processing performed on the second access unit of the second set is performed is an absolute value or a difference value from the time indicated by the first time information. 